On Greedy Algorithms for dictionary with bounded cumulative 

coherence 

Eugene Livshitz 
Abstract 

We discuss the upper and lower estimates for the rate of convergence of Pure and 
Orthogonal Greedy Algorithms for dictionary with bounded cumulative coherence. 

d\ ' 

Introduction. Let H be a real, separable Hilbert space equipped with an inner product 
(SJ ; (•, •) and the norm || • || = (•, •) 1/2 . We say that a set V, V C H is a dictionary if 

> ' 

O ! 9 £ => \\g\\ = 1, and span£> = H. 

Recently the following problem has been intensively studied in Approximation Theory and 
Numeral Analysis: to construct by element / G H and meNan m-term combination 

■ 

£ : / - l> fc (/)<fc(/), c k (f) G R, g k (f) G V 

-i— > . 

that provides a good approximation for /. Greedy Algorithms turn out to be effective for 
i i j obtaining such m-term approximations (see tutorial [T] for details). Two most popular 

of them are defined below. 

Pure Greedy Algorithm (PGA) Set f[ GA := f eH, G£ GA (f,V) := 0. For each 
m > we inductively find g^+i £ ^ su °h that 

\<fF A ,9%t)\ = *w\(fZ 1A ,9)\ (1) 

^ : 

' and define 

O r PGA (f ._ r PGA( f , / ePGA PGA\ PGA 

ePGA /• _ r PGA( e -t-jx _ ePGA _ I rPGA PGA\ PGA 

IT" im+1 - — i "m+l U) y i ~ Jm Wm > 3m+l /Wm+1 ■ 

Orthogonal Greedy Algorithm (OGA) Se£ / OGA ■= f eH, G OGA (f,V) ■= 0. 
For eac/i m>0 we inductively find g^+i G £> suc/i £/iat 

l(/^^i)l = sup|(^ GA ,o)| (2) 

and define 

Gg% A (f,V) := Pro Jg ocA_ g oc + A(f), 

fOGA ._ / rtOGAi t ^\ 
im+1 ■-/~ b m+lU) i; i' 

Thus for f e H and each m > 1 we construct m-term approximations Gf^ A (f,V) 
a ndG° GA (f,V). 
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In this article we study the rate of convergence of Greedy Algorithms for class Ao(T>) 
that is a set of finite linear combination of elements from T> and classes A P (V), 1 < p < 2, 
defined below. For M > we define 

A P (V,M) := {J2^x9 x : < MP, c x el, g x G V, tfA < oo}, 

AeA AeA 

(where closure is taken in the norm of H). Set 

A p (p) := |J ^ P (P,M), 

Af>0 

|/|p := |/|^(z>) := mf{M > : / G .4 P (Z>, M)}, / G ^(P). 
From results of R.A. DeVore, V.N. Temlyakov and E.D. Livshitz |DT], |LTj . [L] it 
follows that Orthogonal Greedy Algorithm does provide the optimal rate of convergence 
d/lim -1 / 2 in A l (T>), but Pure Greedy Algorithm doesn't. For narrower classes such as 
Ao(T>) the rate of convergence of OGA could not be better than CmT x l 2 and would not 
be optimal. In the same time if dictionary D satisfies some additional properties the rate 
of convergence of Greedy Algorithms (for some classes) could be essentially better. This 
area is called Sparse Approximation and has been intensively studied last time ( [GMS] . 
[GN] . |Trj . |DET] ). In this article results will be formulated using the notion of cumulative 
coherence of the dictionary introduced by J. Tropp [TV] 

Mi(P):= sup \&9)\- (3) 

Above-mentioned articles contain the following basic results of Sparse Approximation 
Theory. 

Theorem A. Let V be a dictionary with fii(T>) < 1/2 and f G Aq{T>). Then 

G° GA (f,V) = f, m>m , 
11/ - G? GA (f,V)\\ = \\f m \\ < <7exp(-c(/)m), m > 0. 

For dictionaries with small PGA provides optimal rate of convergence in A P (TJ), 

l<p<2. 

Theorem 1. Let T> be a dictionary with /ii(T>) < 1/3 and f G A\{T>). Then 

||/-G^ A (/,P)|| = \\f m \\ < \f\ im ^ 2 , m>0. 

Theorem 2. Suppose T> is a dictionary with Hi{T>) < 1/3, p, 1 < p < 2 and f G A P (V). 
Then there exist C\ = C\(p) > and C% = C2(fii{T>)) > such that for any m > 1 

\\f-G™ A (f,V)\\ = \\f m \\ < dCal/lpm-^/a. 

In the same time for big (but finite) values of Hi(T>) Pure Greedy Algorithms can 
not always provide exponential rate of convergence, moreover one could be worse than 
Cm- 1 / 2 : 

Theorem 3. There exists a dictionary V with Hi(V) < oo, f G Aq{V), (3 > and C > 
and that such for any m > 1 we have 

\\f a -G p m GA {h,V)\\ = \\f m \\>Cm- l l 2+ P. 
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Properties of dictionaries with bounded cumulative coherence. It's easy to see 
that any dictionary with bounded cumulative coherence in separable Hilbert space is 
countable. Suppose that elements of dictionary are enumerated: V = {g x }\<=n- 

Lemma 1. Let V be a dictionary with ^(V) < 1/2, N > 0, c u E R, g v E V, 1 < v < N. 
Then the following inequalities 

2 



N 



;i-2 / i 1 (p))^ c ^< 



N 

E 

u=l 



c v g 



N 



< 



(1 + 2^(2?)) ^ c 



u=l 



hold. 



Proof. Without loss of generality we can assume that 

|ci| > |c 2 | > • • • > |cjv|. 

We have 



N 

E 

u=l 



c v g 



N 



N 



N 



u=l u=l 



u=l 



Using ([3]) and monotony of \c u \ we estimate 



cl{g»,g v ) + 2c v c^g\g^)-cl 



< 2c„ 



T)=V+l 



< 



Hence 



N 

E 



c v g" 



N 

E> 



<2cl J] \{g\g*)\ <2clfi 1 (D). 



N 



< 



□ 



Lemma 2. Suppose A C N is a finite set of indexes and e > 0. If for f the representation 

f = f e + Y J Cxg\ c x ER, g x EV, (4) 



AeA 



\\fe\\ < e, 



AeA 



(5) 
(6) 



holds, then for A E A we have 

\{f,g X °) ~ c\ \ < f^i(P) max|c A | + e, 



and for X (jL A 



AeA 



\(f,g Xo ) \ < //i(P)max|c A | +e. 
1 AeA 
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Proof. Using representation (j4]) we write for Ao G A 

(f,g x °) - c Xo = (f e + J2 c ^\9 Xo ) - cx (g X( \g Xo ) = £ (cxg\g Xa ) + (fe,g x °) 

AeA AeA, A^A 

and for Ao G' A 

(/, <? A °> = (fe + 5><A <7 A °> = ( C ^\g X °) + (fe,9 X °). 

AeA AeA, A^A 

To complete the proof we estimate using ([6]) and Cauchy - Bunyakovsky - Schwarz in- 
equality 



£ (cxg x ,g Xo ) + (f„g Xo ) 

AeA, A^Aq 



<max|c A | V |(^^ Ao )l + ll/Jlk Ao ll < 
AeA z — ' 

AeA, A^A 

<max|c A | \(g,g Xo )\ + e < max|c A | + e. 

AeA * — ' AeA 



□ 

Lemma 3. Lei T> be a dictionary with fi\(T>) < 1/3, f G A P (T>) m > 1. Assume that 
for n = m — 1, finite AcN and e > i/ie following representation 

f n = f -G p n GA U,V) = f e + J2c x , n g x } c x , n eR, g x eV, \\f t \\ < e (7) 

AeA 

holds. If 

e < -(1 - 3//i(P))max|c Am _i|, (8) 
o AeA 

i/ien /or n = m we get |?P with the same A ; / e and 

Km| p < £ |cA, m -i| p - 2- p (l - 3 Ml (I?)) p max \c x , m ^, 



AeA AeA 



max I c A , m I < max I c A , m -i I . (9) 
AeA AeA 

Proof. From the definition of PGA it follows, that for m > 1 

Therefore it's sufficient to prove the lemma for arbitrary / G A P (V) and m = 1. 

To reduce the notations we write c A instead of c A , A G A. Taking into account (jH]) we 
have 

(l-2/ii(£>))max|c A |-2e > (1 -2>u x (V)) max |c A | -3e > -(l-3/i X (P)) max |c A | > (10) 
AeA AeA 2 AeA 

By Lemma [2] we get 

max I (/, g x ) I > max |c A | — fii(T>) max |c A | — e = (1 — /ijfP)) max |c A | — e, 
AeA AeA AeA AeA 
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for A ^ A, using also ([TO]) we obtain 

\{f,9 X )\ < Vi(P) max |c A | + e < (1 - A*i(X>)) max \c\\ - e. 

AeA AeA 

Therefore there exists Ao G A such that 

| (/, g x °)\ = sup | (/, g) | > (1 - ii X {V)) max |c A | - e. (11) 

Using Lemma [21 we have 

\(f,g Xo )\ < \c Xo \ +/i 1 (P)max|c A | +e. 

AeA 

Combining last two inequalities, we obtain 

(1 - HiCD)) max \c\\ - e < |c Ao | + fJ>i(D) max |c A | + e. 
AeA AeA 

|c Ao | > (1 -2^ 1 (X>))max|c A | - 2e. 

AG A 

Without loss of generality we can assume that c Ao > 0, that is 

c Ao > (l-2 Ml (P))max|c A |-2e. (12) 

AG A 

Applying Lemma [21 ([T2l and (fTDl . we obtain 

(/,fl' Ao ) > c Ao -Hi(T>) max |c A | - e > (1 -3/ii(X>)) max|c A | - 3e > ~(1 - 3/ii(X>)) max|c A |, 

AeA AeA 2 AeA 



(/> £ ) < c Ao + /ii (£>) max | c A | + e. 

AeA 



Hence by $T2]) and (HDD 

Ca - (/, S'* ) > c Ao - ( c Ao + //i(X>) max |c A | + e ) > - ( max |c A | + e ) > 



(13) 



AeA J \ AeA 

> -Ca +(1 - 3/ii(P))max|c A | - 3e > -c Ao + -(1 - 3//i(X>)) max |c A |. (14) 

AeA 2 AeA 

Combining ([TB"1) and ([H]), we estimate 

K - (/, <? A °>| + ^(1 - 3/n(P)) max| CA | < c Ao , 

^ AGA 



|CA - (f,9 Xo )\ p < cl Q - Q(l - 3/^(2))) max|c A |) 



V/' 



(15) 



If we set 

c A ,i = c A = c A)0 , A e A \ {A }, 
ca ,i = c Ao ,o - (f,9 X °), 

then statement of the lemma will folow from (fT5|) . □ 



Proof of Theorem [T], Lemma [3] implies that for any m > 

|/Ji<|/|i- 

Using Lemma 3.5 from |DT] and Lemma [3] we have for m > 

\(fm,9m+i)\ = swp\(f m ,g}\ > > "' / '" 



geP l/mll l/ll 

By definition of PGA 

||/ m+ l|| 2 = ||/ m |r - (fm,9m + l) 2 < \\fm\\ 2 ~ ( ^ V = ||/ m f ( 1 - 



l/li 7 V l/l? 

Applying Lemma 3.4 from [DTJ for a m = ||/ m _i|| 2 and A = \f\l and taking into account 
the inequality 

ai = ll/o|| 2 <l/li, 
we obtain that for {a m }^ =1 such that 

a m +i < a m ^1 - lyjlj , a%<A, 

the following inequality 

a m < Am^ 1 

holds. Thus for m > 1 we have 

ll/ m ||=^+i<l/|i(^ + l)" 1/2 <l/|i^ 1/2 - 
This completes the proof of the theorem. 

Proof of Theorem [2]. Let k > 1 and / G *4. p . For arbitrary e 

0<e<i(l-3 /Ul (D))A;- 1 ^|/| p , (16) 
o 

there exists representation (jl]) such that inequalities (jSJ) and hold. We claim that there 
exists n, < n < k such that 

max | CA , n | < c 1 (p)c 2 (jJii(p))k- 1 / p \f\ J „ (17) 

Afc/v 

$> A ,„r < 5> A , | P = !/!£• (18) 
AeA AeA 

For every m — 1, . . . , k forn = m— 1 the representation (j7j) hold (beginning with ca,o : = c A ) 
and either 

-i| < K~ 1/p | 



max |c A>m -i| S ft ' l/lp, 

AfrA. 



in this case we can set n — m — 1, or 



max |c A , m -i| < fc 1/p |/| 
AeA 



6 



p- 



Then taking into account (|T6j) we have (151) . Therefore, by Lemma [3] the representation 
d7j) holds for n = m and using fl^D and ([2]) we have 



< V |c A , m | p < Y] \cx,m-i\ p - 2~ p (l - BmWmax \c x>m ^\ p = • • • = 
AeA AeA 
m 

= V|c Ai o| p -V2-f(l-3 Ml (P))fmax|c A)m _ 1 ^ < |/g-m2~P (l-3 Ml (P))^max |c A>m p 
* — ' z — ' AeA ' AeA 

AeA n=l 

max |c A , m | < 2(1 - 3 f i 1 (V))- 1 m- 1 '*\f\ p , ^ |c A)m p> < |/|* 



1/2 

+ e < 



AeA 

This provides (fl7|) and (Tl8|) for n = k. 
Using (IPTj) and (|T8|) . we estimate 

^ |cA,n| 2 = |cA,n| P |cA,n| 2_P = f max |c A , n | J '^'^ - 

AeA AeA ^ ' AeA 

< (c,{ P ) Ci {^)k^\ft P ) l/l^ = C3(p)c 4 (/ii(P)fc- 2/p+1 |/lp- 

Applying Lemma [1] and ([7j), we obtain 

IIMI < H/nll < ||EC A ^ A || + ||/ £ || < ( (1 + 2/^))^ 4," J 

AeA \ AeA / 

Since e > can be arbitrary small the last inequality completes the proof of Theorem [2j 
□ 
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